{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import scipy as sp\n",
    "import numpy.linalg\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib as mpl\n",
    "import sympy as sy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.set_printoptions(precision=3)\n",
    "np.set_printoptions(suppress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def round_expr(expr, num_digits):\n",
    "    return expr.xreplace({n : round(n, num_digits) for n in expr.atoms(sy.Number)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font face=\"gotham\" color=\"purple\"> Diagonalization of Symmetric Matrices</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The <font face=\"gotham\" color=\"red\"> first</font>  theorem of symmetric matrix:\n",
    "\n",
    "<font face=\"gotham\" color=\"red\">If $A$ is symmetric, i.e. $A = A^T$, then any two eigenvectors from different eigenspaces are orthogonal.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\begin{aligned}\n",
    "\\lambda_{1} \\mathbf{v}_{1} \\cdot \\mathbf{v}_{2} &=\\left(\\lambda_{1} \\mathbf{v}_{1}\\right)^{T} \\mathbf{v}_{2}=\\left(A \\mathbf{v}_{1}\\right)^{T} \\mathbf{v}_{2} \\\\\n",
    "&=\\left(\\mathbf{v}_{1}^{T} A^{T}\\right) \\mathbf{v}_{2}=\\mathbf{v}_{1}^{T}\\left(A \\mathbf{v}_{2}\\right) \\\\\n",
    "&=\\mathbf{v}_{1}^{T}\\left(\\lambda_{2} \\mathbf{v}_{2}\\right) \\\\\n",
    "&=\\lambda_{2} \\mathbf{v}_{1}^{T} \\mathbf{v}_{2}=\\lambda_{2} \\mathbf{v}_{1} \\cdot \\mathbf{v}_{2}\n",
    "\\end{aligned}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because $\\lambda_1 \\neq \\lambda_2$, so only condition which makes the equation holds is \n",
    "\n",
    "$$\n",
    " \\mathbf{v}_{1} \\cdot \\mathbf{v}_{2}=0\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the help of this theorem, we can conclude that if symmetric matrix $A$ has different eigenvalues, its corresponding eigenvectors must be mutually orthogonal.\n",
    "\n",
    "The diagonalization of $A$ is \n",
    "\n",
    "$$\n",
    "A = PDP^T = PDP^{-1}\n",
    "$$\n",
    "\n",
    "where $P$ is an orthonormal matrix with all eigenvectors of $A$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The <font face=\"gotham\" color=\"red\"> second</font> theorem of symmetric matrix:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font face=\"gotham\" color=\"red\"> An $n \\times n$ matrix $A$ is orthogonally diagonalizable if and only if $A$ is a symmetric matrix: $A^{T}=\\left(P D P^{T}\\right)^{T}=P^{T T} D^{T} P^{T}=P D P^{T}=A$.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> An Example</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a random symmetric matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 2., 2.],\n",
       "       [1., 0., 2.],\n",
       "       [1., 2., 1.]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = np.round(2*np.random.rand(3, 3)); A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[8., 4., 6.],\n",
       "       [4., 5., 3.],\n",
       "       [6., 3., 6.]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "B = A@A.T; B # generate a symmetric matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform diagonalization with ```np.linalg.eig()```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.709,  0.699,  0.089],\n",
       "       [ 0.399, -0.294, -0.868],\n",
       "       [ 0.581, -0.651,  0.488]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "D, P = np.linalg.eig(B); P"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[11.787,  0.   ,  0.   ],\n",
       "       [ 0.   ,  0.082,  0.   ],\n",
       "       [ 0.   ,  0.   ,  4.131]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "D = np.diag(D); D"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the norm of all eigenvectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9999999999999998\n",
      "0.9999999999999999\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "for i in [0, 1, 2]:\n",
    "    print(np.linalg.norm(P[:,i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the orthogonality of eigenvectors, see if $PP^T=I$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 0.],\n",
       "       [0., 1., 0.],\n",
       "       [0., 0., 1.]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P@P.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> The Spectral Theorem</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An $n \\times n$ symmetric matrix $A$ has the following properties:\n",
    "1. $A$ has $n$ real eigenvalues, counting multiplicities.\n",
    "2. The dimension of the eigenspace for each eigenvalue $\\lambda$ equals the multiplicity of $\\lambda$ as a root of the characteristic equation.\n",
    "3. The eigenspaces are mutually orthogonal, in the sense that eigenvectors corresponding to different eigenvalues are orthogonal.\n",
    "4. $A$ is orthogonally diagonalizable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All these properties are obvious without proof, as the example above shows.However the purpose of the theorem  is not reiterating last section, it paves the way for <font face=\"gotham\" color=\"red\">spectral decomposition</font>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write diagonalization explicitly, we get the representation of spectral decomposition\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "A &=P D P^{T}=\\left[\\begin{array}{lll}\n",
    "\\mathbf{u}_{1} & \\cdots & \\mathbf{u}_{n}\n",
    "\\end{array}\\right]\\left[\\begin{array}{ccc}\n",
    "\\lambda_{1} & & 0 \\\\\n",
    "& \\ddots & \\\\\n",
    "0 & & \\lambda_{n}\n",
    "\\end{array}\\right]\\left[\\begin{array}{c}\n",
    "\\mathbf{u}_{1}^{T} \\\\\n",
    "\\vdots \\\\\n",
    "\\mathbf{u}_{n}^{T}\n",
    "\\end{array}\\right] \\\\\n",
    "&=\\left[\\begin{array}{lll}\n",
    "\\lambda_{1} \\mathbf{u}_{1} & \\cdots & \\lambda_{n} \\mathbf{u}_{n}\n",
    "\\end{array}\\right]\\left[\\begin{array}{c}\n",
    "\\mathbf{u}_{1}^{T} \\\\\n",
    "\\vdots \\\\\n",
    "\\mathbf{u}_{n}^{T}\n",
    "\\end{array}\\right]\\\\\n",
    "&= \\lambda_{1} \\mathbf{u}_{1} \\mathbf{u}_{1}^{T}+\\lambda_{2} \\mathbf{u}_{2} \\mathbf{u}_{2}^{T}+\\cdots+\\lambda_{n} \\mathbf{u}_{n} \\mathbf{u}_{n}^{T}\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$ \\mathbf{u}_{i} \\mathbf{u}_{i}^{T}$ are rank $1$ symmetric matrices, because all rows of $ \\mathbf{u}_{i} \\mathbf{u}_{i}^{T}$ are multiples of $\\mathbf{u}_{i}^{T}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following the example above, we demonstrate in SymPy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "lamb0,lamb1,lamb2 = D[0,0], D[1,1], D[2,2]\n",
    "u0,u1,u2 = P[:,0], P[:,1], P[:,2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check rank of $ \\mathbf{u}_{i} \\mathbf{u}_{i}^{T}$ by ```np.linalg.matrix_rank()```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.matrix_rank(np.outer(u0,u0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use spectral theorem to recover $A$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2., 3., 2.],\n",
       "       [3., 6., 4.],\n",
       "       [2., 4., 8.]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "specDecomp = lamb0 * np.outer(u0,u0) + lamb1 * np.outer(u1,u1) + lamb2 * np.outer(u2,u2)\n",
    "specDecomp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font face=\"gotham\" color=\"purple\"> Quadratic Form</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A <font face=\"gotham\" color=\"red\"> quadratic form</font> is a function with form $Q(\\mathbf{x})=\\mathbf{x}^TA\\mathbf{x}$, where $A$ is an $n\\times n$ symmetric matrix, which is called the <font face=\"gotham\" color=\"red\"> the matrix of the quadratic form</font>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider a matrix of quadratic form \n",
    "\n",
    "$$\n",
    "A = \n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "3 & 2 & 0\\\\\n",
    "2 & -1 & 4\\\\\n",
    "0 & 4 & -2\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "$$\n",
    "\n",
    "construct the quadratic form $\\mathbf{x}^TA\\mathbf{x}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "\\mathbf{x}^TA\\mathbf{x}&=\n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "x_1 & x_2 & x_3\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "3 & 2 & 0\\\\\n",
    "2 & -1 & 4\\\\\n",
    "0 & 4 & -2\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "x_1 \\\\ x_2\\\\ x_3\n",
    "\\end{matrix}\n",
    "\\right]\\\\\n",
    "& =\\left[\n",
    "\\begin{matrix}\n",
    "x_1 & x_2 & x_3\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "3x_1+2x_2 \\\\ 2x_1-x_2+4x_3 \\\\ 4x_2-2x_3\n",
    "\\end{matrix}\n",
    "\\right]\\\\\n",
    "& = \n",
    "x_1(3x_1+2x_2)+x_2(2x_1-x_2+4x_3)+x_3(4x_2-2x_3)\\\\\n",
    "& = 3x_1^2+4x_1x_2-x_2^2+8x_2x_3-2x_3^2\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fortunately, there is an easier way to calculate quadratic form.\n",
    "\n",
    "Notice that coefficients of $x_i^2$ is on the principal diagonal and coefficients of $x_ix_j$ are be split evenly between $(i,j)-$ and $(j, i)-$entries in $A$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Example </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider another example,\n",
    "\n",
    "$$\n",
    "A = \n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "3 & 2 & 0 & 5\\\\\n",
    "2 & -1 & 4 & -3\\\\\n",
    "0 & 4 & -2 & -4\\\\\n",
    "5 & -3 & -4 & 7\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All $x_i^2$'s terms are \n",
    "\n",
    "$$\n",
    "3x_1^2-x_2^2-2x_3^2+7x_4^2\n",
    "$$\n",
    "\n",
    "whose coefficients are from principal diagonal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All $x_ix_j$'s terms are \n",
    "\n",
    "$$\n",
    "4x_1x_2+0x_1x_3+10x_1x_4+8x_2x_3-6x_2x_4-8x_3x_4\n",
    "$$\n",
    "\n",
    "Add up together then quadratic form is \n",
    "\n",
    "$$\n",
    "3x_1^2-x_2^2-2x_3^2+7x_4^2+4x_1x_2+0x_1x_3+10x_1x_4+8x_2x_3-6x_2x_4-8x_3x_4\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's verify in SymPy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "x1, x2, x3, x4 = sy.symbols('x_1 x_2 x_3 x_4')\n",
    "A = sy.Matrix([[3,2,0,5],[2,-1,4,-3],[0,4,-2,-4],[5,-3,-4,7]])\n",
    "x = sy.Matrix([x1, x2, x3, x4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\left[\\begin{matrix}3 x_{1}^{2} + 4 x_{1} x_{2} + 10 x_{1} x_{4} - x_{2}^{2} + 8 x_{2} x_{3} - 6 x_{2} x_{4} - 2 x_{3}^{2} - 8 x_{3} x_{4} + 7 x_{4}^{2}\\end{matrix}\\right]$"
      ],
      "text/plain": [
       "Matrix([[3*x_1**2 + 4*x_1*x_2 + 10*x_1*x_4 - x_2**2 + 8*x_2*x_3 - 6*x_2*x_4 - 2*x_3**2 - 8*x_3*x_4 + 7*x_4**2]])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sy.expand(x.T*A*x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results is exactly the same as we derived."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Change of Variable in Quadratic Forms</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To convert a matrix of quadratic form into diagonal matrix can save us same troubles, that is to say, no cross products terms. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since $A$ is symmetric, there is an orthonormal $P$ that\n",
    "\n",
    "$$\n",
    "PDP^T = A \\qquad \\text{and}\\qquad PP^T = I\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can show that\n",
    "\n",
    "$$\n",
    "\\mathbf{x}^TA\\mathbf{x}=\\mathbf{x}^TIAI\\mathbf{x}=\\mathbf{x}^TPP^TAPP^T\\mathbf{x}=\\mathbf{x}^TPDP^T\\mathbf{x}=(P^T\\mathbf{x})^TDP^T\\mathbf{x}=\\mathbf{y}^T D \\mathbf{y}$$\n",
    "\n",
    "where $P^T$ defined a coordinate transformation and $\\mathbf{y} = P^T\\mathbf{x}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider $A$\n",
    "\n",
    "$$\n",
    "A = \n",
    "\\left[\n",
    "\\begin{matrix}\n",
    "3 & 2 & 0\\\\\n",
    "2 & -1 & 4\\\\\n",
    "0 & 4 & -2\n",
    "\\end{matrix}\n",
    "\\right]\n",
    "$$\n",
    "\n",
    "Find eigenvalue and eigenvectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 3,  2,  0],\n",
       "       [ 2, -1,  4],\n",
       "       [ 0,  4, -2]])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = np.array([[3,2,0],[2,-1,4],[0,4,-2]]); A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 4.388,  0.   ,  0.   ],\n",
       "       [ 0.   ,  1.35 ,  0.   ],\n",
       "       [ 0.   ,  0.   , -5.738]])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "D, P = np.linalg.eig(A)\n",
    "D = np.diag(D); D"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test if $P$ is normalized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1., -0., -0.],\n",
       "       [-0.,  1., -0.],\n",
       "       [-0., -0.,  1.]])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P.T@P"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compute $\\mathbf{y}= P^T\\mathbf{x}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\left[\\begin{matrix}x_{1}\\\\x_{2}\\\\x_{3}\\end{matrix}\\right]$"
      ],
      "text/plain": [
       "Matrix([\n",
       "[x1],\n",
       "[x2],\n",
       "[x3]])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1, x2, x3 = sy.symbols('x1 x2 x3')\n",
    "x = sy.Matrix([[x1], [x2], [x3]])\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\left[\\begin{matrix}0.7738 & -0.6143 & 0.1544\\\\0.5369 & 0.5067 & -0.6746\\\\0.3362 & 0.6049 & 0.7219\\end{matrix}\\right]$"
      ],
      "text/plain": [
       "Matrix([\n",
       "[0.7738, -0.6143,  0.1544],\n",
       "[0.5369,  0.5067, -0.6746],\n",
       "[0.3362,  0.6049,  0.7219]])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P = round_expr(sy.Matrix(P), 4); P"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the $\\mathbf{y} = P^T \\mathbf{x}$ is \n",
    "\n",
    "$$\n",
    "\\left[\\begin{matrix}0.7738 x_{1} + 0.5369 x_{2} + 0.3362 x_{3}\\\\- 0.6143 x_{1} + 0.5067 x_{2} + 0.6049 x_{3}\\\\0.1544 x_{1} - 0.6746 x_{2} + 0.7219 x_{3}\\end{matrix}\\right]\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The transformed quadratic form $\\mathbf{y}^T D \\mathbf{y}$ is"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\left[\\begin{matrix}4.3876 & 0.0 & 0.0\\\\0.0 & 1.3505 & 0.0\\\\0.0 & 0.0 & -5.7381\\end{matrix}\\right]$"
      ],
      "text/plain": [
       "Matrix([\n",
       "[4.3876,    0.0,     0.0],\n",
       "[   0.0, 1.3505,     0.0],\n",
       "[   0.0,    0.0, -5.7381]])"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "D =  round_expr(sy.Matrix(D),4);D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\left[\\begin{matrix}y_{1}\\\\y_{2}\\\\y_{3}\\end{matrix}\\right]$"
      ],
      "text/plain": [
       "Matrix([\n",
       "[y1],\n",
       "[y2],\n",
       "[y3]])"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y1, y2, y3 = sy.symbols('y1 y2 y3')\n",
    "y = sy.Matrix([[y1], [y2], [y3]]);y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/latex": [
       "$\\displaystyle \\left[\\begin{matrix}4.3876 y_{1}^{2} + 1.3505 y_{2}^{2} - 5.7381 y_{3}^{2}\\end{matrix}\\right]$"
      ],
      "text/plain": [
       "Matrix([[4.3876*y1**2 + 1.3505*y2**2 - 5.7381*y3**2]])"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.T*D*y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Visualize the Quadratic Form</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The codes are exceedingly lengthy, but intuitive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "/* Put everything inside the global mpl namespace */\n",
       "window.mpl = {};\n",
       "\n",
       "\n",
       "mpl.get_websocket_type = function() {\n",
       "    if (typeof(WebSocket) !== 'undefined') {\n",
       "        return WebSocket;\n",
       "    } else if (typeof(MozWebSocket) !== 'undefined') {\n",
       "        return MozWebSocket;\n",
       "    } else {\n",
       "        alert('Your browser does not have WebSocket support. ' +\n",
       "              'Please try Chrome, Safari or Firefox ≥ 6. ' +\n",
       "              'Firefox 4 and 5 are also supported but you ' +\n",
       "              'have to enable WebSockets in about:config.');\n",
       "    };\n",
       "}\n",
       "\n",
       "mpl.figure = function(figure_id, websocket, ondownload, parent_element) {\n",
       "    this.id = figure_id;\n",
       "\n",
       "    this.ws = websocket;\n",
       "\n",
       "    this.supports_binary = (this.ws.binaryType != undefined);\n",
       "\n",
       "    if (!this.supports_binary) {\n",
       "        var warnings = document.getElementById(\"mpl-warnings\");\n",
       "        if (warnings) {\n",
       "            warnings.style.display = 'block';\n",
       "            warnings.textContent = (\n",
       "                \"This browser does not support binary websocket messages. \" +\n",
       "                    \"Performance may be slow.\");\n",
       "        }\n",
       "    }\n",
       "\n",
       "    this.imageObj = new Image();\n",
       "\n",
       "    this.context = undefined;\n",
       "    this.message = undefined;\n",
       "    this.canvas = undefined;\n",
       "    this.rubberband_canvas = undefined;\n",
       "    this.rubberband_context = undefined;\n",
       "    this.format_dropdown = undefined;\n",
       "\n",
       "    this.image_mode = 'full';\n",
       "\n",
       "    this.root = $('<div/>');\n",
       "    this._root_extra_style(this.root)\n",
       "    this.root.attr('style', 'display: inline-block');\n",
       "\n",
       "    $(parent_element).append(this.root);\n",
       "\n",
       "    this._init_header(this);\n",
       "    this._init_canvas(this);\n",
       "    this._init_toolbar(this);\n",
       "\n",
       "    var fig = this;\n",
       "\n",
       "    this.waiting = false;\n",
       "\n",
       "    this.ws.onopen =  function () {\n",
       "            fig.send_message(\"supports_binary\", {value: fig.supports_binary});\n",
       "            fig.send_message(\"send_image_mode\", {});\n",
       "            if (mpl.ratio != 1) {\n",
       "                fig.send_message(\"set_dpi_ratio\", {'dpi_ratio': mpl.ratio});\n",
       "            }\n",
       "            fig.send_message(\"refresh\", {});\n",
       "        }\n",
       "\n",
       "    this.imageObj.onload = function() {\n",
       "            if (fig.image_mode == 'full') {\n",
       "                // Full images could contain transparency (where diff images\n",
       "                // almost always do), so we need to clear the canvas so that\n",
       "                // there is no ghosting.\n",
       "                fig.context.clearRect(0, 0, fig.canvas.width, fig.canvas.height);\n",
       "            }\n",
       "            fig.context.drawImage(fig.imageObj, 0, 0);\n",
       "        };\n",
       "\n",
       "    this.imageObj.onunload = function() {\n",
       "        fig.ws.close();\n",
       "    }\n",
       "\n",
       "    this.ws.onmessage = this._make_on_message_function(this);\n",
       "\n",
       "    this.ondownload = ondownload;\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._init_header = function() {\n",
       "    var titlebar = $(\n",
       "        '<div class=\"ui-dialog-titlebar ui-widget-header ui-corner-all ' +\n",
       "        'ui-helper-clearfix\"/>');\n",
       "    var titletext = $(\n",
       "        '<div class=\"ui-dialog-title\" style=\"width: 100%; ' +\n",
       "        'text-align: center; padding: 3px;\"/>');\n",
       "    titlebar.append(titletext)\n",
       "    this.root.append(titlebar);\n",
       "    this.header = titletext[0];\n",
       "}\n",
       "\n",
       "\n",
       "\n",
       "mpl.figure.prototype._canvas_extra_style = function(canvas_div) {\n",
       "\n",
       "}\n",
       "\n",
       "\n",
       "mpl.figure.prototype._root_extra_style = function(canvas_div) {\n",
       "\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._init_canvas = function() {\n",
       "    var fig = this;\n",
       "\n",
       "    var canvas_div = $('<div/>');\n",
       "\n",
       "    canvas_div.attr('style', 'position: relative; clear: both; outline: 0');\n",
       "\n",
       "    function canvas_keyboard_event(event) {\n",
       "        return fig.key_event(event, event['data']);\n",
       "    }\n",
       "\n",
       "    canvas_div.keydown('key_press', canvas_keyboard_event);\n",
       "    canvas_div.keyup('key_release', canvas_keyboard_event);\n",
       "    this.canvas_div = canvas_div\n",
       "    this._canvas_extra_style(canvas_div)\n",
       "    this.root.append(canvas_div);\n",
       "\n",
       "    var canvas = $('<canvas/>');\n",
       "    canvas.addClass('mpl-canvas');\n",
       "    canvas.attr('style', \"left: 0; top: 0; z-index: 0; outline: 0\")\n",
       "\n",
       "    this.canvas = canvas[0];\n",
       "    this.context = canvas[0].getContext(\"2d\");\n",
       "\n",
       "    var backingStore = this.context.backingStorePixelRatio ||\n",
       "\tthis.context.webkitBackingStorePixelRatio ||\n",
       "\tthis.context.mozBackingStorePixelRatio ||\n",
       "\tthis.context.msBackingStorePixelRatio ||\n",
       "\tthis.context.oBackingStorePixelRatio ||\n",
       "\tthis.context.backingStorePixelRatio || 1;\n",
       "\n",
       "    mpl.ratio = (window.devicePixelRatio || 1) / backingStore;\n",
       "\n",
       "    var rubberband = $('<canvas/>');\n",
       "    rubberband.attr('style', \"position: absolute; left: 0; top: 0; z-index: 1;\")\n",
       "\n",
       "    var pass_mouse_events = true;\n",
       "\n",
       "    canvas_div.resizable({\n",
       "        start: function(event, ui) {\n",
       "            pass_mouse_events = false;\n",
       "        },\n",
       "        resize: function(event, ui) {\n",
       "            fig.request_resize(ui.size.width, ui.size.height);\n",
       "        },\n",
       "        stop: function(event, ui) {\n",
       "            pass_mouse_events = true;\n",
       "            fig.request_resize(ui.size.width, ui.size.height);\n",
       "        },\n",
       "    });\n",
       "\n",
       "    function mouse_event_fn(event) {\n",
       "        if (pass_mouse_events)\n",
       "            return fig.mouse_event(event, event['data']);\n",
       "    }\n",
       "\n",
       "    rubberband.mousedown('button_press', mouse_event_fn);\n",
       "    rubberband.mouseup('button_release', mouse_event_fn);\n",
       "    // Throttle sequential mouse events to 1 every 20ms.\n",
       "    rubberband.mousemove('motion_notify', mouse_event_fn);\n",
       "\n",
       "    rubberband.mouseenter('figure_enter', mouse_event_fn);\n",
       "    rubberband.mouseleave('figure_leave', mouse_event_fn);\n",
       "\n",
       "    canvas_div.on(\"wheel\", function (event) {\n",
       "        event = event.originalEvent;\n",
       "        event['data'] = 'scroll'\n",
       "        if (event.deltaY < 0) {\n",
       "            event.step = 1;\n",
       "        } else {\n",
       "            event.step = -1;\n",
       "        }\n",
       "        mouse_event_fn(event);\n",
       "    });\n",
       "\n",
       "    canvas_div.append(canvas);\n",
       "    canvas_div.append(rubberband);\n",
       "\n",
       "    this.rubberband = rubberband;\n",
       "    this.rubberband_canvas = rubberband[0];\n",
       "    this.rubberband_context = rubberband[0].getContext(\"2d\");\n",
       "    this.rubberband_context.strokeStyle = \"#000000\";\n",
       "\n",
       "    this._resize_canvas = function(width, height) {\n",
       "        // Keep the size of the canvas, canvas container, and rubber band\n",
       "        // canvas in synch.\n",
       "        canvas_div.css('width', width)\n",
       "        canvas_div.css('height', height)\n",
       "\n",
       "        canvas.attr('width', width * mpl.ratio);\n",
       "        canvas.attr('height', height * mpl.ratio);\n",
       "        canvas.attr('style', 'width: ' + width + 'px; height: ' + height + 'px;');\n",
       "\n",
       "        rubberband.attr('width', width);\n",
       "        rubberband.attr('height', height);\n",
       "    }\n",
       "\n",
       "    // Set the figure to an initial 600x600px, this will subsequently be updated\n",
       "    // upon first draw.\n",
       "    this._resize_canvas(600, 600);\n",
       "\n",
       "    // Disable right mouse context menu.\n",
       "    $(this.rubberband_canvas).bind(\"contextmenu\",function(e){\n",
       "        return false;\n",
       "    });\n",
       "\n",
       "    function set_focus () {\n",
       "        canvas.focus();\n",
       "        canvas_div.focus();\n",
       "    }\n",
       "\n",
       "    window.setTimeout(set_focus, 100);\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._init_toolbar = function() {\n",
       "    var fig = this;\n",
       "\n",
       "    var nav_element = $('<div/>');\n",
       "    nav_element.attr('style', 'width: 100%');\n",
       "    this.root.append(nav_element);\n",
       "\n",
       "    // Define a callback function for later on.\n",
       "    function toolbar_event(event) {\n",
       "        return fig.toolbar_button_onclick(event['data']);\n",
       "    }\n",
       "    function toolbar_mouse_event(event) {\n",
       "        return fig.toolbar_button_onmouseover(event['data']);\n",
       "    }\n",
       "\n",
       "    for(var toolbar_ind in mpl.toolbar_items) {\n",
       "        var name = mpl.toolbar_items[toolbar_ind][0];\n",
       "        var tooltip = mpl.toolbar_items[toolbar_ind][1];\n",
       "        var image = mpl.toolbar_items[toolbar_ind][2];\n",
       "        var method_name = mpl.toolbar_items[toolbar_ind][3];\n",
       "\n",
       "        if (!name) {\n",
       "            // put a spacer in here.\n",
       "            continue;\n",
       "        }\n",
       "        var button = $('<button/>');\n",
       "        button.addClass('ui-button ui-widget ui-state-default ui-corner-all ' +\n",
       "                        'ui-button-icon-only');\n",
       "        button.attr('role', 'button');\n",
       "        button.attr('aria-disabled', 'false');\n",
       "        button.click(method_name, toolbar_event);\n",
       "        button.mouseover(tooltip, toolbar_mouse_event);\n",
       "\n",
       "        var icon_img = $('<span/>');\n",
       "        icon_img.addClass('ui-button-icon-primary ui-icon');\n",
       "        icon_img.addClass(image);\n",
       "        icon_img.addClass('ui-corner-all');\n",
       "\n",
       "        var tooltip_span = $('<span/>');\n",
       "        tooltip_span.addClass('ui-button-text');\n",
       "        tooltip_span.html(tooltip);\n",
       "\n",
       "        button.append(icon_img);\n",
       "        button.append(tooltip_span);\n",
       "\n",
       "        nav_element.append(button);\n",
       "    }\n",
       "\n",
       "    var fmt_picker_span = $('<span/>');\n",
       "\n",
       "    var fmt_picker = $('<select/>');\n",
       "    fmt_picker.addClass('mpl-toolbar-option ui-widget ui-widget-content');\n",
       "    fmt_picker_span.append(fmt_picker);\n",
       "    nav_element.append(fmt_picker_span);\n",
       "    this.format_dropdown = fmt_picker[0];\n",
       "\n",
       "    for (var ind in mpl.extensions) {\n",
       "        var fmt = mpl.extensions[ind];\n",
       "        var option = $(\n",
       "            '<option/>', {selected: fmt === mpl.default_extension}).html(fmt);\n",
       "        fmt_picker.append(option);\n",
       "    }\n",
       "\n",
       "    // Add hover states to the ui-buttons\n",
       "    $( \".ui-button\" ).hover(\n",
       "        function() { $(this).addClass(\"ui-state-hover\");},\n",
       "        function() { $(this).removeClass(\"ui-state-hover\");}\n",
       "    );\n",
       "\n",
       "    var status_bar = $('<span class=\"mpl-message\"/>');\n",
       "    nav_element.append(status_bar);\n",
       "    this.message = status_bar[0];\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.request_resize = function(x_pixels, y_pixels) {\n",
       "    // Request matplotlib to resize the figure. Matplotlib will then trigger a resize in the client,\n",
       "    // which will in turn request a refresh of the image.\n",
       "    this.send_message('resize', {'width': x_pixels, 'height': y_pixels});\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.send_message = function(type, properties) {\n",
       "    properties['type'] = type;\n",
       "    properties['figure_id'] = this.id;\n",
       "    this.ws.send(JSON.stringify(properties));\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.send_draw_message = function() {\n",
       "    if (!this.waiting) {\n",
       "        this.waiting = true;\n",
       "        this.ws.send(JSON.stringify({type: \"draw\", figure_id: this.id}));\n",
       "    }\n",
       "}\n",
       "\n",
       "\n",
       "mpl.figure.prototype.handle_save = function(fig, msg) {\n",
       "    var format_dropdown = fig.format_dropdown;\n",
       "    var format = format_dropdown.options[format_dropdown.selectedIndex].value;\n",
       "    fig.ondownload(fig, format);\n",
       "}\n",
       "\n",
       "\n",
       "mpl.figure.prototype.handle_resize = function(fig, msg) {\n",
       "    var size = msg['size'];\n",
       "    if (size[0] != fig.canvas.width || size[1] != fig.canvas.height) {\n",
       "        fig._resize_canvas(size[0], size[1]);\n",
       "        fig.send_message(\"refresh\", {});\n",
       "    };\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_rubberband = function(fig, msg) {\n",
       "    var x0 = msg['x0'] / mpl.ratio;\n",
       "    var y0 = (fig.canvas.height - msg['y0']) / mpl.ratio;\n",
       "    var x1 = msg['x1'] / mpl.ratio;\n",
       "    var y1 = (fig.canvas.height - msg['y1']) / mpl.ratio;\n",
       "    x0 = Math.floor(x0) + 0.5;\n",
       "    y0 = Math.floor(y0) + 0.5;\n",
       "    x1 = Math.floor(x1) + 0.5;\n",
       "    y1 = Math.floor(y1) + 0.5;\n",
       "    var min_x = Math.min(x0, x1);\n",
       "    var min_y = Math.min(y0, y1);\n",
       "    var width = Math.abs(x1 - x0);\n",
       "    var height = Math.abs(y1 - y0);\n",
       "\n",
       "    fig.rubberband_context.clearRect(\n",
       "        0, 0, fig.canvas.width / mpl.ratio, fig.canvas.height / mpl.ratio);\n",
       "\n",
       "    fig.rubberband_context.strokeRect(min_x, min_y, width, height);\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_figure_label = function(fig, msg) {\n",
       "    // Updates the figure title.\n",
       "    fig.header.textContent = msg['label'];\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_cursor = function(fig, msg) {\n",
       "    var cursor = msg['cursor'];\n",
       "    switch(cursor)\n",
       "    {\n",
       "    case 0:\n",
       "        cursor = 'pointer';\n",
       "        break;\n",
       "    case 1:\n",
       "        cursor = 'default';\n",
       "        break;\n",
       "    case 2:\n",
       "        cursor = 'crosshair';\n",
       "        break;\n",
       "    case 3:\n",
       "        cursor = 'move';\n",
       "        break;\n",
       "    }\n",
       "    fig.rubberband_canvas.style.cursor = cursor;\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_message = function(fig, msg) {\n",
       "    fig.message.textContent = msg['message'];\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_draw = function(fig, msg) {\n",
       "    // Request the server to send over a new figure.\n",
       "    fig.send_draw_message();\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_image_mode = function(fig, msg) {\n",
       "    fig.image_mode = msg['mode'];\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.updated_canvas_event = function() {\n",
       "    // Called whenever the canvas gets updated.\n",
       "    this.send_message(\"ack\", {});\n",
       "}\n",
       "\n",
       "// A function to construct a web socket function for onmessage handling.\n",
       "// Called in the figure constructor.\n",
       "mpl.figure.prototype._make_on_message_function = function(fig) {\n",
       "    return function socket_on_message(evt) {\n",
       "        if (evt.data instanceof Blob) {\n",
       "            /* FIXME: We get \"Resource interpreted as Image but\n",
       "             * transferred with MIME type text/plain:\" errors on\n",
       "             * Chrome.  But how to set the MIME type?  It doesn't seem\n",
       "             * to be part of the websocket stream */\n",
       "            evt.data.type = \"image/png\";\n",
       "\n",
       "            /* Free the memory for the previous frames */\n",
       "            if (fig.imageObj.src) {\n",
       "                (window.URL || window.webkitURL).revokeObjectURL(\n",
       "                    fig.imageObj.src);\n",
       "            }\n",
       "\n",
       "            fig.imageObj.src = (window.URL || window.webkitURL).createObjectURL(\n",
       "                evt.data);\n",
       "            fig.updated_canvas_event();\n",
       "            fig.waiting = false;\n",
       "            return;\n",
       "        }\n",
       "        else if (typeof evt.data === 'string' && evt.data.slice(0, 21) == \"data:image/png;base64\") {\n",
       "            fig.imageObj.src = evt.data;\n",
       "            fig.updated_canvas_event();\n",
       "            fig.waiting = false;\n",
       "            return;\n",
       "        }\n",
       "\n",
       "        var msg = JSON.parse(evt.data);\n",
       "        var msg_type = msg['type'];\n",
       "\n",
       "        // Call the  \"handle_{type}\" callback, which takes\n",
       "        // the figure and JSON message as its only arguments.\n",
       "        try {\n",
       "            var callback = fig[\"handle_\" + msg_type];\n",
       "        } catch (e) {\n",
       "            console.log(\"No handler for the '\" + msg_type + \"' message type: \", msg);\n",
       "            return;\n",
       "        }\n",
       "\n",
       "        if (callback) {\n",
       "            try {\n",
       "                // console.log(\"Handling '\" + msg_type + \"' message: \", msg);\n",
       "                callback(fig, msg);\n",
       "            } catch (e) {\n",
       "                console.log(\"Exception inside the 'handler_\" + msg_type + \"' callback:\", e, e.stack, msg);\n",
       "            }\n",
       "        }\n",
       "    };\n",
       "}\n",
       "\n",
       "// from http://stackoverflow.com/questions/1114465/getting-mouse-location-in-canvas\n",
       "mpl.findpos = function(e) {\n",
       "    //this section is from http://www.quirksmode.org/js/events_properties.html\n",
       "    var targ;\n",
       "    if (!e)\n",
       "        e = window.event;\n",
       "    if (e.target)\n",
       "        targ = e.target;\n",
       "    else if (e.srcElement)\n",
       "        targ = e.srcElement;\n",
       "    if (targ.nodeType == 3) // defeat Safari bug\n",
       "        targ = targ.parentNode;\n",
       "\n",
       "    // jQuery normalizes the pageX and pageY\n",
       "    // pageX,Y are the mouse positions relative to the document\n",
       "    // offset() returns the position of the element relative to the document\n",
       "    var x = e.pageX - $(targ).offset().left;\n",
       "    var y = e.pageY - $(targ).offset().top;\n",
       "\n",
       "    return {\"x\": x, \"y\": y};\n",
       "};\n",
       "\n",
       "/*\n",
       " * return a copy of an object with only non-object keys\n",
       " * we need this to avoid circular references\n",
       " * http://stackoverflow.com/a/24161582/3208463\n",
       " */\n",
       "function simpleKeys (original) {\n",
       "  return Object.keys(original).reduce(function (obj, key) {\n",
       "    if (typeof original[key] !== 'object')\n",
       "        obj[key] = original[key]\n",
       "    return obj;\n",
       "  }, {});\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.mouse_event = function(event, name) {\n",
       "    var canvas_pos = mpl.findpos(event)\n",
       "\n",
       "    if (name === 'button_press')\n",
       "    {\n",
       "        this.canvas.focus();\n",
       "        this.canvas_div.focus();\n",
       "    }\n",
       "\n",
       "    var x = canvas_pos.x * mpl.ratio;\n",
       "    var y = canvas_pos.y * mpl.ratio;\n",
       "\n",
       "    this.send_message(name, {x: x, y: y, button: event.button,\n",
       "                             step: event.step,\n",
       "                             guiEvent: simpleKeys(event)});\n",
       "\n",
       "    /* This prevents the web browser from automatically changing to\n",
       "     * the text insertion cursor when the button is pressed.  We want\n",
       "     * to control all of the cursor setting manually through the\n",
       "     * 'cursor' event from matplotlib */\n",
       "    event.preventDefault();\n",
       "    return false;\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._key_event_extra = function(event, name) {\n",
       "    // Handle any extra behaviour associated with a key event\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.key_event = function(event, name) {\n",
       "\n",
       "    // Prevent repeat events\n",
       "    if (name == 'key_press')\n",
       "    {\n",
       "        if (event.which === this._key)\n",
       "            return;\n",
       "        else\n",
       "            this._key = event.which;\n",
       "    }\n",
       "    if (name == 'key_release')\n",
       "        this._key = null;\n",
       "\n",
       "    var value = '';\n",
       "    if (event.ctrlKey && event.which != 17)\n",
       "        value += \"ctrl+\";\n",
       "    if (event.altKey && event.which != 18)\n",
       "        value += \"alt+\";\n",
       "    if (event.shiftKey && event.which != 16)\n",
       "        value += \"shift+\";\n",
       "\n",
       "    value += 'k';\n",
       "    value += event.which.toString();\n",
       "\n",
       "    this._key_event_extra(event, name);\n",
       "\n",
       "    this.send_message(name, {key: value,\n",
       "                             guiEvent: simpleKeys(event)});\n",
       "    return false;\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.toolbar_button_onclick = function(name) {\n",
       "    if (name == 'download') {\n",
       "        this.handle_save(this, null);\n",
       "    } else {\n",
       "        this.send_message(\"toolbar_button\", {name: name});\n",
       "    }\n",
       "};\n",
       "\n",
       "mpl.figure.prototype.toolbar_button_onmouseover = function(tooltip) {\n",
       "    this.message.textContent = tooltip;\n",
       "};\n",
       "mpl.toolbar_items = [[\"Home\", \"Reset original view\", \"fa fa-home icon-home\", \"home\"], [\"Back\", \"Back to previous view\", \"fa fa-arrow-left icon-arrow-left\", \"back\"], [\"Forward\", \"Forward to next view\", \"fa fa-arrow-right icon-arrow-right\", \"forward\"], [\"\", \"\", \"\", \"\"], [\"Pan\", \"Pan axes with left mouse, zoom with right\", \"fa fa-arrows icon-move\", \"pan\"], [\"Zoom\", \"Zoom to rectangle\", \"fa fa-square-o icon-check-empty\", \"zoom\"], [\"\", \"\", \"\", \"\"], [\"Download\", \"Download plot\", \"fa fa-floppy-o icon-save\", \"download\"]];\n",
       "\n",
       "mpl.extensions = [\"eps\", \"jpeg\", \"pdf\", \"png\", \"ps\", \"raw\", \"svg\", \"tif\"];\n",
       "\n",
       "mpl.default_extension = \"png\";var comm_websocket_adapter = function(comm) {\n",
       "    // Create a \"websocket\"-like object which calls the given IPython comm\n",
       "    // object with the appropriate methods. Currently this is a non binary\n",
       "    // socket, so there is still some room for performance tuning.\n",
       "    var ws = {};\n",
       "\n",
       "    ws.close = function() {\n",
       "        comm.close()\n",
       "    };\n",
       "    ws.send = function(m) {\n",
       "        //console.log('sending', m);\n",
       "        comm.send(m);\n",
       "    };\n",
       "    // Register the callback with on_msg.\n",
       "    comm.on_msg(function(msg) {\n",
       "        //console.log('receiving', msg['content']['data'], msg);\n",
       "        // Pass the mpl event to the overridden (by mpl) onmessage function.\n",
       "        ws.onmessage(msg['content']['data'])\n",
       "    });\n",
       "    return ws;\n",
       "}\n",
       "\n",
       "mpl.mpl_figure_comm = function(comm, msg) {\n",
       "    // This is the function which gets called when the mpl process\n",
       "    // starts-up an IPython Comm through the \"matplotlib\" channel.\n",
       "\n",
       "    var id = msg.content.data.id;\n",
       "    // Get hold of the div created by the display call when the Comm\n",
       "    // socket was opened in Python.\n",
       "    var element = $(\"#\" + id);\n",
       "    var ws_proxy = comm_websocket_adapter(comm)\n",
       "\n",
       "    function ondownload(figure, format) {\n",
       "        window.open(figure.imageObj.src);\n",
       "    }\n",
       "\n",
       "    var fig = new mpl.figure(id, ws_proxy,\n",
       "                           ondownload,\n",
       "                           element.get(0));\n",
       "\n",
       "    // Call onopen now - mpl needs it, as it is assuming we've passed it a real\n",
       "    // web socket which is closed, not our websocket->open comm proxy.\n",
       "    ws_proxy.onopen();\n",
       "\n",
       "    fig.parent_element = element.get(0);\n",
       "    fig.cell_info = mpl.find_output_cell(\"<div id='\" + id + \"'></div>\");\n",
       "    if (!fig.cell_info) {\n",
       "        console.error(\"Failed to find cell for figure\", id, fig);\n",
       "        return;\n",
       "    }\n",
       "\n",
       "    var output_index = fig.cell_info[2]\n",
       "    var cell = fig.cell_info[0];\n",
       "\n",
       "};\n",
       "\n",
       "mpl.figure.prototype.handle_close = function(fig, msg) {\n",
       "    var width = fig.canvas.width/mpl.ratio\n",
       "    fig.root.unbind('remove')\n",
       "\n",
       "    // Update the output cell to use the data from the current canvas.\n",
       "    fig.push_to_output();\n",
       "    var dataURL = fig.canvas.toDataURL();\n",
       "    // Re-enable the keyboard manager in IPython - without this line, in FF,\n",
       "    // the notebook keyboard shortcuts fail.\n",
       "    IPython.keyboard_manager.enable()\n",
       "    $(fig.parent_element).html('<img src=\"' + dataURL + '\" width=\"' + width + '\">');\n",
       "    fig.close_ws(fig, msg);\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.close_ws = function(fig, msg){\n",
       "    fig.send_message('closing', msg);\n",
       "    // fig.ws.close()\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.push_to_output = function(remove_interactive) {\n",
       "    // Turn the data on the canvas into data in the output cell.\n",
       "    var width = this.canvas.width/mpl.ratio\n",
       "    var dataURL = this.canvas.toDataURL();\n",
       "    this.cell_info[1]['text/html'] = '<img src=\"' + dataURL + '\" width=\"' + width + '\">';\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.updated_canvas_event = function() {\n",
       "    // Tell IPython that the notebook contents must change.\n",
       "    IPython.notebook.set_dirty(true);\n",
       "    this.send_message(\"ack\", {});\n",
       "    var fig = this;\n",
       "    // Wait a second, then push the new image to the DOM so\n",
       "    // that it is saved nicely (might be nice to debounce this).\n",
       "    setTimeout(function () { fig.push_to_output() }, 1000);\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._init_toolbar = function() {\n",
       "    var fig = this;\n",
       "\n",
       "    var nav_element = $('<div/>');\n",
       "    nav_element.attr('style', 'width: 100%');\n",
       "    this.root.append(nav_element);\n",
       "\n",
       "    // Define a callback function for later on.\n",
       "    function toolbar_event(event) {\n",
       "        return fig.toolbar_button_onclick(event['data']);\n",
       "    }\n",
       "    function toolbar_mouse_event(event) {\n",
       "        return fig.toolbar_button_onmouseover(event['data']);\n",
       "    }\n",
       "\n",
       "    for(var toolbar_ind in mpl.toolbar_items){\n",
       "        var name = mpl.toolbar_items[toolbar_ind][0];\n",
       "        var tooltip = mpl.toolbar_items[toolbar_ind][1];\n",
       "        var image = mpl.toolbar_items[toolbar_ind][2];\n",
       "        var method_name = mpl.toolbar_items[toolbar_ind][3];\n",
       "\n",
       "        if (!name) { continue; };\n",
       "\n",
       "        var button = $('<button class=\"btn btn-default\" href=\"#\" title=\"' + name + '\"><i class=\"fa ' + image + ' fa-lg\"></i></button>');\n",
       "        button.click(method_name, toolbar_event);\n",
       "        button.mouseover(tooltip, toolbar_mouse_event);\n",
       "        nav_element.append(button);\n",
       "    }\n",
       "\n",
       "    // Add the status bar.\n",
       "    var status_bar = $('<span class=\"mpl-message\" style=\"text-align:right; float: right;\"/>');\n",
       "    nav_element.append(status_bar);\n",
       "    this.message = status_bar[0];\n",
       "\n",
       "    // Add the close button to the window.\n",
       "    var buttongrp = $('<div class=\"btn-group inline pull-right\"></div>');\n",
       "    var button = $('<button class=\"btn btn-mini btn-primary\" href=\"#\" title=\"Stop Interaction\"><i class=\"fa fa-power-off icon-remove icon-large\"></i></button>');\n",
       "    button.click(function (evt) { fig.handle_close(fig, {}); } );\n",
       "    button.mouseover('Stop Interaction', toolbar_mouse_event);\n",
       "    buttongrp.append(button);\n",
       "    var titlebar = this.root.find($('.ui-dialog-titlebar'));\n",
       "    titlebar.prepend(buttongrp);\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._root_extra_style = function(el){\n",
       "    var fig = this\n",
       "    el.on(\"remove\", function(){\n",
       "\tfig.close_ws(fig, {});\n",
       "    });\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._canvas_extra_style = function(el){\n",
       "    // this is important to make the div 'focusable\n",
       "    el.attr('tabindex', 0)\n",
       "    // reach out to IPython and tell the keyboard manager to turn it's self\n",
       "    // off when our div gets focus\n",
       "\n",
       "    // location in version 3\n",
       "    if (IPython.notebook.keyboard_manager) {\n",
       "        IPython.notebook.keyboard_manager.register_events(el);\n",
       "    }\n",
       "    else {\n",
       "        // location in version 2\n",
       "        IPython.keyboard_manager.register_events(el);\n",
       "    }\n",
       "\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._key_event_extra = function(event, name) {\n",
       "    var manager = IPython.notebook.keyboard_manager;\n",
       "    if (!manager)\n",
       "        manager = IPython.keyboard_manager;\n",
       "\n",
       "    // Check for shift+enter\n",
       "    if (event.shiftKey && event.which == 13) {\n",
       "        this.canvas_div.blur();\n",
       "        // select the cell after this one\n",
       "        var index = IPython.notebook.find_cell_index(this.cell_info[0]);\n",
       "        IPython.notebook.select(index + 1);\n",
       "    }\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_save = function(fig, msg) {\n",
       "    fig.ondownload(fig, null);\n",
       "}\n",
       "\n",
       "\n",
       "mpl.find_output_cell = function(html_output) {\n",
       "    // Return the cell and output element which can be found *uniquely* in the notebook.\n",
       "    // Note - this is a bit hacky, but it is done because the \"notebook_saving.Notebook\"\n",
       "    // IPython event is triggered only after the cells have been serialised, which for\n",
       "    // our purposes (turning an active figure into a static one), is too late.\n",
       "    var cells = IPython.notebook.get_cells();\n",
       "    var ncells = cells.length;\n",
       "    for (var i=0; i<ncells; i++) {\n",
       "        var cell = cells[i];\n",
       "        if (cell.cell_type === 'code'){\n",
       "            for (var j=0; j<cell.output_area.outputs.length; j++) {\n",
       "                var data = cell.output_area.outputs[j];\n",
       "                if (data.data) {\n",
       "                    // IPython >= 3 moved mimebundle to data attribute of output\n",
       "                    data = data.data;\n",
       "                }\n",
       "                if (data['text/html'] == html_output) {\n",
       "                    return [cell, data, j];\n",
       "                }\n",
       "            }\n",
       "        }\n",
       "    }\n",
       "}\n",
       "\n",
       "// Register the function which deals with the matplotlib target/channel.\n",
       "// The kernel may be null if the page has been refreshed.\n",
       "if (IPython.notebook.kernel != null) {\n",
       "    IPython.notebook.kernel.comm_manager.register_target('matplotlib', mpl.mpl_figure_comm);\n",
       "}\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<img src=\"\" width=\"700\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib notebook\n",
    "k = 6\n",
    "x = np.linspace(-k, k)\n",
    "y = np.linspace(-k, k)\n",
    "X, Y = np.meshgrid(x, y)\n",
    "\n",
    "fig = plt.figure(figsize = (7, 7))\n",
    "\n",
    "########################### xAx 1 ############################\n",
    "Z = 3*X**2 + 7*Y**2\n",
    "ax = fig.add_subplot(221, projection='3d')\n",
    "ax.plot_wireframe(X, Y, Z, linewidth = 1.5, alpha = .3, color = 'r')\n",
    "ax.set_title('$z = 3x^2+7y^2$')\n",
    "\n",
    "xarrow = np.array([[-5, 0, 0, 10, 0, 0]])\n",
    "X1, Y1, Z1, U1, V1, W1 = zip(*xarrow)\n",
    "ax.quiver(X1, Y1, Z1, U1, V1, W1, length=1, normalize=False, color = 'black', \n",
    "          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',\n",
    "          linestyles = 'solid',linewidths = 2)\n",
    "\n",
    "yarrow = np.array([[0, -5, 0, 0, 10, 0]])\n",
    "X2, Y2, Z2, U2, V2, W2 = zip(*yarrow)\n",
    "ax.quiver(X2, Y2, Z2, U2, V2, W2, length=1, normalize=False, color = 'black',\n",
    "          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',\n",
    "          linestyles = 'solid',linewidths = 2)\n",
    "\n",
    "zarrow = np.array([[0, 0, -3, 0, 0, 300]])\n",
    "X3, Y3, Z3, U3, V3, W3 = zip(*zarrow)\n",
    "ax.quiver(X3, Y3, Z3, U3, V3, W3, length=1, normalize=False, color = 'black', \n",
    "alpha = .6, arrow_length_ratio = .001, pivot = 'tail',\n",
    "          linestyles = 'solid',linewidths = 2)\n",
    "\n",
    "########################### xAx 2 ############################\n",
    "Z = 3*X**2\n",
    "ax = fig.add_subplot(222, projection='3d')\n",
    "ax.plot_wireframe(X, Y, Z, linewidth = 1.5, alpha = .3, color = 'r')\n",
    "ax.set_title('$z = 3x^2$')\n",
    "xarrow = np.array([[-5, 0, 0, 10, 0, 0]])\n",
    "X1, Y1, Z1, U1, V1, W1 = zip(*xarrow)\n",
    "ax.quiver(X1, Y1, Z1, U1, V1, W1, length=1, normalize=False, color = 'black', \n",
    "          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',\n",
    "          linestyles = 'solid',linewidths = 2)\n",
    "\n",
    "yarrow = np.array([[0, -5, 0, 0, 10, 0]])\n",
    "X2, Y2, Z2, U2, V2, W2 = zip(*yarrow)\n",
    "ax.quiver(X2, Y2, Z2, U2, V2, W2, length=1, normalize=False, color = 'black', \n",
    "          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',\n",
    "          linestyles = 'solid',linewidths = 2)\n",
    "\n",
    "zarrow = np.array([[0, 0, -3, 0, 0, 800]])\n",
    "X3, Y3, Z3, U3, V3, W3 = zip(*zarrow)\n",
    "ax.quiver(X3, Y3, Z3, U3, V3, W3, length=1, normalize=False, color = 'black', \n",
    "          alpha = .6, arrow_length_ratio = .001, pivot = 'tail',\n",
    "          linestyles = 'solid',linewidths = 2)\n",
    "\n",
    "########################### xAx 3 ############################\n",
    "Z = 3*X**2 - 7*Y**2\n",
    "ax = fig.add_subplot(223, projection='3d')\n",
    "ax.plot_wireframe(X, Y, Z, linewidth = 1.5, alpha = .3, color = 'r')\n",
    "ax.set_title('$z = 3x^2-7y^2$')\n",
    "xarrow = np.array([[-5, 0, 0, 10, 0, 0]])\n",
    "X1, Y1, Z1, U1, V1, W1 = zip(*xarrow)\n",
    "ax.quiver(X1, Y1, Z1, U1, V1, W1, length=1, normalize=False, color = 'black', \n",
    "          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',\n",
    "          linestyles = 'solid',linewidths = 2)\n",
    "\n",
    "yarrow = np.array([[0, -5, 0, 0, 10, 0]])\n",
    "X2, Y2, Z2, U2, V2, W2 = zip(*yarrow)\n",
    "ax.quiver(X2, Y2, Z2, U2, V2, W2, length=1, normalize=False, color = 'black', \n",
    "          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',\n",
    "          linestyles = 'solid',linewidths = 2)\n",
    "\n",
    "zarrow = np.array([[0, 0, -150, 0, 0, 300]])\n",
    "X3, Y3, Z3, U3, V3, W3 = zip(*zarrow)\n",
    "ax.quiver(X3, Y3, Z3, U3, V3, W3, length=1, normalize=False, color = 'black', \n",
    "          alpha = .6, arrow_length_ratio = .001, pivot = 'tail',\n",
    "          linestyles = 'solid',linewidths = 2)\n",
    "\n",
    "########################### xAx 4 ############################\n",
    "Z = -3*X**2 - 7*Y**2\n",
    "ax = fig.add_subplot(224, projection='3d')\n",
    "ax.plot_wireframe(X, Y, Z, linewidth = 1.5, alpha = .3, color = 'r')\n",
    "ax.set_title('$z = -3x^2-7y^2$')\n",
    "xarrow = np.array([[-5, 0, 0, 10, 0, 0]])\n",
    "X1, Y1, Z1, U1, V1, W1 = zip(*xarrow)\n",
    "ax.quiver(X1, Y1, Z1, U1, V1, W1, length=1, normalize=False, color = 'black', \n",
    "          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',\n",
    "          linestyles = 'solid',linewidths = 2)\n",
    "\n",
    "yarrow = np.array([[0, -5, 0, 0, 10, 0]])\n",
    "X2, Y2, Z2, U2, V2, W2 = zip(*yarrow)\n",
    "ax.quiver(X2, Y2, Z2, U2, V2, W2, length=1, normalize=False, color = 'black', \n",
    "          alpha = .6, arrow_length_ratio = .12, pivot = 'tail',\n",
    "          linestyles = 'solid',linewidths = 2)\n",
    "\n",
    "zarrow = np.array([[0, 0, -300, 0, 0, 330]])\n",
    "X3, Y3, Z3, U3, V3, W3 = zip(*zarrow)\n",
    "ax.quiver(X3, Y3, Z3, U3, V3, W3, length=1, normalize=False, color = 'black', \n",
    "          alpha = .6, arrow_length_ratio = .001, pivot = 'tail',\n",
    "          linestyles = 'solid',linewidths = 2)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now there are some important terms to define, a quadratic form $Q$ is:\n",
    "1. positive definite if $Q(\\mathbf{x})>0$ for all $\\mathbf{x} \\neq \\mathbf{0}$\n",
    "2. negative definite if $Q(\\mathbf{x})<0$ for all $\\mathbf{x} \\neq \\mathbf{0}$\n",
    "3. positive semidefinite if $Q(\\mathbf{x})\\geq0$ for all $\\mathbf{x} \\neq \\mathbf{0}$\n",
    "4. negative semidefinite if $Q(\\mathbf{x})\\leq0$ for all $\\mathbf{x} \\neq \\mathbf{0}$\n",
    "5. indefinite if $Q(\\mathbf{x})$ assumes both positive and negative values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a theorem for quadratic forms and eigenvalues:\n",
    "\n",
    "Let $A$ be an $n \\times n$ symmetric matrix. Then a quadratic form $\\mathbf{x}^{T} A \\mathbf{x}$ is:\n",
    "\n",
    "\n",
    "1. positive definite if and only if the eigenvalues of $A$ are all positive\n",
    "2. negative definite if and only if the eigenvalues of $A$ are all negative\n",
    "3. indefinite if and only if $A$ has both positive and negative eigenvalues\n",
    "\n",
    "With the help of this theorem, we can immediate tell if a quadratic form has a maximum, minimum or saddle point after calculating the eigenvalues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Positive Definite Matrix</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Symmetric matrices are one of most important kinds in linear algrebra, we will show they are always positive definite.\n",
    "\n",
    "${A}$ is a symmetric matrix, premultiplying ${A}\\mathbf{x}=\\lambda \\mathbf{x}$ by $\\mathbf{x}^T$\n",
    "\n",
    "$$\n",
    "\\mathbf{x}^T{A}\\mathbf{x} = \\lambda \\mathbf{x}^T\\mathbf{x} = \\lambda \\|\\mathbf{x}\\|^2\n",
    "$$\n",
    "\n",
    "$\\mathbf{x}^T{A}\\mathbf{x}$ must be postive, since we defined so, then $\\lambda$ must be larger than $0$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try asking the other way around: if all eigenvalues are positive, is $A_{n\\times n}$ positive definite? Yes.\n",
    "\n",
    "Here is the <font face=\"gotham\" color=\"red\">Principal Axes Theorem</font> which employs the orthogonal change of variable $\\mathbf{x}=P\\mathbf{y}$:\n",
    "\n",
    "$$\n",
    "Q(\\mathbf{x})=\\mathbf{x}^{T} A \\mathbf{x}=\\mathbf{y}^{T} D \\mathbf{y}=\\lambda_{1} y_{1}^{2}+\\lambda_{2} y_{2}^{2}+\\cdots+\\lambda_{n} y_{n}^{2}\n",
    "$$\n",
    "\n",
    "If all of $\\lambda$'s are positive, $\\mathbf{x}^{T} A \\mathbf{x}$ is also positive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\"> Cholesky Decomposition</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cholesky decomposition is modification of $LU$ decomposition. And it is more efficient than $LU$ algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If $A$ is positive definite matrix, i.e. $\\mathbf{x}^{T} A \\mathbf{x}>0$ or every eigenvalue is strictly positive. A positive definite matrix can be decomposed into a multiplication of lower triagnluar matrix and its transpose."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\begin{aligned}\n",
    "{A}={L} {L}^{T} &=\\left[\\begin{array}{ccc}\n",
    "l_{11} & 0 & 0 \\\\\n",
    "l_{21} & l_{22} & 0 \\\\\n",
    "l_{31} & l_{32} & l_{33}\n",
    "\\end{array}\\right]\\left[\\begin{array}{ccc}\n",
    "l_{11} & l_{21} & l_{31} \\\\\n",
    "0 & l_{22} & l_{32} \\\\\n",
    "0 & 0 & l_{33}\n",
    "\\end{array}\\right] \\\\\n",
    "\\left[\\begin{array}{ccc}\n",
    "a_{11} & a_{21} & a_{31} \\\\\n",
    "a_{21} & a_{22} & a_{32} \\\\\n",
    "a_{31} & a_{32} & a_{33}\n",
    "\\end{array}\\right] \n",
    "&=\\left[\\begin{array}{ccc}\n",
    "l_{11}^{2} &l_{21} l_{11} & l_{31} l_{11} \\\\\n",
    "l_{21} l_{11} & l_{21}^{2}+l_{22}^{2} & l_{31} l_{21}+l_{32} l_{22} \\\\\n",
    "l_{31} l_{11} & l_{31} l_{21}+l_{32} l_{22} & l_{31}^{2}+l_{32}^{2}+l_{33}^{2}\n",
    "\\end{array}\\right]\n",
    "\\end{aligned}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will show this with NumPy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[16, -8, -4],\n",
       "       [-8, 29, 12],\n",
       "       [-4, 12, 41]])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = np.array([[16, -8, -4], [-8, 29, 12], [-4, 12, 41]]); A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 4.,  0.,  0.],\n",
       "       [-2.,  5.,  0.],\n",
       "       [-1.,  2.,  6.]])"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L = sp.linalg.cholesky(A, lower = True); L"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check if $LL^T=A$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[16., -8., -4.],\n",
       "       [-8., 29., 12.],\n",
       "       [-4., 12., 41.]])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L@L.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font face=\"gotham\" color=\"purple\"> Some Facts of Symmetric Matrices</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font face=\"gotham\" color=\"purple\">Rank and Positive Definiteness</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a symmetric matrix $A$ does not have full rank, which means there must be a non-trivial vector $\\mathbf{v}$ satisfies\n",
    "\n",
    "$$\n",
    "A\\mathbf{v} = \\mathbf{0}\n",
    "$$\n",
    "\n",
    "which also means the quadratic form equals zero $\\mathbf{v}^TA\\mathbf{v} = \\mathbf{0}$. Thus $A$ can not be a positive definite matrix if it does not have full rank."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contrarily, a matrix to be positive definite must have full rank."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "toc-autonumbering": true,
  "toc-showcode": false,
  "toc-showmarkdowntxt": true,
  "toc-showtags": true
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
